Lips-Sync 3D Speech Animation using Compact Key-Shapes

Facial animation is traditionally considered as important but tedious work for most applications, because the muscles on face are complex and dynamically interacting. Although there are several methods proposed to ease the burden from artists to create animating faces, non of these are fast and efficient in storage. This paper introduces a framework for synthesizing lips-sync facial animation given a speech sequence. Starting from tracking features on training videos, the method first find representative keyshapes that is important for both image reconstruction and guiding the artists to create corresponding 3D models. The training video is then parameterized to weighting space, or cross-mapping, then the dynamic of features on the face is learned for each kind of phoneme. The propose system can synthesis lips-sync 3D facial animation in very short time, and requires very small amount storage to keep information of the key-shape models and phoneme dynamics.